Add v2.1 data #50

forsyth2 · 2024-11-13T00:55:21Z

Add v2.1 data. Resolves #48.

forsyth2 · 2024-11-13T02:07:26Z

@chengzhuzhang The way I have this set up now:

The HPSS path (and consequently the data size) column is empty because /home/projects/e3sm/www/WaterCycle/E3SMv2.1/LR/ doesn't seem to exist on NERSC HPSS. I saw your email about copying data to /home/projects/e3sm/www/WaterCycle/E3SMv2_1/, so I imagine this will work once that data is copied.
The ESGF links don't seem to be set up correctly though. They're giving "No Data". E.g., for piControl: https://esgf-node.llnl.gov/search/cmip6/?source_id=E3SM-2.1-0&experiment_id=piControl&variant_label=r1i1p1f1 (CMIP) and https://esgf-node.llnl.gov/search/e3sm/?model_version=2.1_0&experiment=piControl&ensemble_member=ens1 (Native). What needs to be changed in those link paths?
I suppressed the creation of a reproduction script table since we only have the original data here.
Should we also add the original run scripts to a v2.1 equivalent of https://github.com/E3SM-Project/e3sm_data_docs/tree/main/run_scripts/v2/original? Will those scripts be in the HPSS paths too?

chengzhuzhang · 2024-11-14T17:59:37Z

@forsyth2 I restarted copying data after Perlmutter gets back online. Yes, the data is located under: /home/projects/e3sm/www/WaterCycle/E3SMv2_1/, I will let you know once the data is in place.

The data is published through ANL's ESGF node. So the pre-fix for the CMIP urls needs to be https://esgf-node.cels.anl.gov/, also the syntax for each urls are changed. Please find the correct per-simulation urls. We don't have native data published, so no Native urls in this case.

The original run scripts can be found here: https://github.com/E3SM-Project/SimulationScripts/tree/master/archive/v2_1. yes, I suppose we don't need reproduction script for now.

forsyth2 · 2024-11-14T23:49:12Z

the data is located under: /home/projects/e3sm/www/WaterCycle/E3SMv2_1/, I will let you know once the data is in place.

I'm not sure how far along this is, but I'm currently getting the following error, so I can't even check its status.

A:/home/f/forsyth-> ls /home/projects/e3sm/www/WaterCycle/E3SMv2   

/home/projects/e3sm/www/WaterCycle/E3SMv2:
LR/     NARRM/  
A:/home/f/forsyth-> ls /home/projects/e3sm/www/WaterCycle/E3SMv2_1 
*** hpss_Opendir: Access denied [-13: HPSS_EACCES] 
    /home/projects/e3sm/www/WaterCycle/E3SMv2_1

/home/projects/e3sm/www/WaterCycle/E3SMv2_1:

chengzhuzhang · 2024-11-15T00:06:25Z

A:/home/f/forsyth-> ls /home/projects/e3sm/www/WaterCycle/E3SMv2_1
*** hpss_Opendir: Access denied [-13: HPSS_EACCES]

I guess this is expected. After all simulations are in place, the permission needs to be reset.

forsyth2 · 2024-11-15T18:48:07Z

@chengzhuzhang Remaining action items after my latest changes, which can be seen at https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_50/html/v2.1/WaterCycle/simulation_data/simulation_table.html

After all simulations are in place, the permission needs to be reset.

They still might be processing, but it looks like I still can't access the HPSS path, which is why the Data Size and HPSS Path columns are currently empty.

Please find the correct per-simulation urls.

Do you have any guidance on finding the URLs? https://esgf-node.cels.anl.gov/search/e3sm/?model_version=2_1 shows a 404 error, as does simply https://esgf-node.cels.anl.gov/search/e3sm.

The original run scripts can be found here: https://github.com/E3SM-Project/SimulationScripts/tree/master/archive/v2_1.

I've copied just the scripts to run_scripts/v2.1/original. These aren't currently linked anywhere though. We could 1) add the directory link on https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_50/html/v2.1/WaterCycle/simulation_data/index.html or 2) add the original scripts column to the simulation data table (rather than the reproduction scripts table, which we aren't including here).

chengzhuzhang · 2024-11-15T19:37:38Z

yes, the data is still in transfer.

Here is the example url for 1pctCO2: https://esgf-node.cels.anl.gov/search/?project=CMIP6&activeFacets=%7B%22source_id%22%3A%22E3SM-2-1%22%2C%22experiment_id%22%3A%221pctCO2%22%7D
For getting the urls for each simulation:
0 Goto https://esgf-node.cels.anl.gov/

click on the box next to CMIP6
in the facet menu, click on Identifies -> select SourceID= E3SM-2-1 ,
which will filter out all the relevant experiments.
You can find the Copy Search on the upper right corner for cpying the url link.
See fig below:

For simulation script, either way is fine.

forsyth2 · 2024-11-15T22:17:47Z

For getting the urls for each simulation:

Thanks, the screenshot with directions was very helpful. I'm still having trouble getting the variant labels to work out correctly, but overall it's looking better. I'll keep trying on that.

For simulation script, either way is fine.

Great, I just added a line of text pointing to the directory.

forsyth2

@chengzhuzhang Please review https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_50/html/v2.1/WaterCycle/simulation_data/index.html and https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_50/html/v2.1/WaterCycle/simulation_data/simulation_table.html.

If those looks good to you, we can merge this.

forsyth2 · 2024-11-18T21:02:14Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+
+The E3SMv2.1 simulation data is available on **ESGF** and **NERSC HPSS**.
+
+The preferred retrieval method is **ESGF**. Native output is available at `ESGF <https://esgf-node.cels.anl.gov/search/e3sm/?model_version=2_1>`_, and a subset of the data is reformatted to conform to CMIP conventions and submmited to the CMIP6 ESGF archive (ESGF links are provided in the table below for published data).


Checked link works ✅

Actually it's https://esgf-node.cels.anl.gov/search?project=CMIP6&activeFacets=%7B%22source_id%22%3A%22E3SM-2-1%22%7D that works; that edit didn't seem to make it into the last commit.

forsyth2 · 2024-11-18T21:02:29Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+
+**v2_1.LR** simulations data has been archived on NERSC HPSS under: ::
+
+  /home/projects/e3sm/www/WaterCycle/E3SMv2_1


Checked HPSS path ✅

forsyth2 · 2024-11-18T21:02:55Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+  /home/projects/e3sm/www/WaterCycle/E3SMv2_1
+
+
+Original run scripts can be found `here <https://github.com/E3SM-Project/e3sm_data_docs/tree/main/run_scripts/v2.1/original>`_.


This link will exist after this PR is merged.

forsyth2 · 2024-11-18T21:04:22Z

utils/simulations_v2_1.csv

@@ -0,0 +1,10 @@
+model_version, group, resolution, category, simulation_name,    machine, checksum, experiment, ensemble_num, cmip_only, node,


All the v2.1 simulations are defined here.

forsyth2 · 2024-11-18T21:04:51Z

utils/generate_tables.py

-    if experiment and ensemble_num:
+def get_esgf(source_id: str, model_version: str, experiment: str, ensemble_num: str, cmip_only: str, node: str) -> str:
+    if node == "cels.anl":
+        esgf = f"`CMIP <https://esgf-node.{node}.gov/search/?project=CMIP6&activeFacets=%7B%22source_id%22%3A%22{source_id}%22%2C%22experiment_id%22%3A%22{experiment}%22%2C%22variant_label%22%3A%22r{ensemble_num}i1p1f1%22%7D>`_"


Checked these links on https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_50/html/v2.1/WaterCycle/simulation_data/simulation_table.html ✅

chengzhuzhang · 2024-11-18T23:10:49Z

docs/source/v2.1/WaterCycle/index.rst

+* **v2_1.LR** (lower resolution)
+
+If you use data from this simulation campaign, please cite the relevant overview
+manuscripts.


Please cite Smith et al. 2024, GMD as the overview reference paper
https://gmd.copernicus.org/preprints/gmd-2024-149/

chengzhuzhang · 2024-11-18T23:12:18Z

docs/source/v2.1/WaterCycle/index.rst

+E3SMv2.1 (Water Cycle)
+====================
+
+The simulation compaign for E3SMv2 (Water Cycle) was performed with two


This needs to be modified as The simulation campaign for E3SMv2.1 was performed with Low Resolution (LR) configuration

chengzhuzhang · 2024-11-18T23:14:16Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+
+or ::
+
+   zstash extract --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/<HPSS path below>


Same as above

chengzhuzhang · 2024-11-18T23:17:47Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+2. To retrieve files with zstash command:
+   ::
+
+    zstash extract --hpss=/home/projects/e3sm/www/WaterCycle/E3SMv2_1/LR/v2.LR.piControl "*.elm.h0.00[3-4]?-??.nc"


The path is not correct. In addition, there is a type elm (_should be eam) inheriting from v2.

chengzhuzhang

Hey @forsyth2
Thank you for the PR, please see my requested changes.
The zstash instruction is mostly copied from v2, and modified to use v2.1 data path. I think it is okay to keep this as is, but we can re-orgnize at a later time, so that we don't need to repeat each time when new simulation (e.g. v3) is added.

forsyth2 · 2024-11-19T01:27:40Z

@chengzhuzhang The latest commit addresses your comments. Let me know how the pages look now:

chengzhuzhang · 2024-11-19T03:44:41Z

Thank you for addressing the comments. The pages looks nicely!

chengzhuzhang · 2024-11-19T19:44:01Z

docs/source/v2.1/WaterCycle/simulation_data/index.rst

+
+The E3SMv2.1 simulation data is available on **ESGF** and **NERSC HPSS**.
+
+The preferred retrieval method is **ESGF**. Native output is available at `ESGF <https://esgf-node.cels.anl.gov/search?project=CMIP6&activeFacets=%7B%22source_id%22%3A%22E3SM-2-1%22%7D>`_, and a subset of the data is reformatted to conform to CMIP conventions and submmited to the CMIP6 ESGF archive (ESGF links are provided in the table below for published data).


@forsyth2
Sorry that I didn't catch this problem, but found it when reviewing the newsletter article. We did not publish v2.1 native data to ESGF. Could you update this to:
The data is reformatted to conform to CMIP conventions and submmited to the CMIP6 ESGF archive (ESGF links are provided in the table below for published data).

forsyth2 self-assigned this Nov 13, 2024

forsyth2 mentioned this pull request Nov 13, 2024

Asd crt/add screamv0 v1 sim data #49

Merged

Add v2.1 data

2212473

forsyth2 force-pushed the issue-48-v2-1-data branch from 0e9e2b8 to 474d69c Compare November 15, 2024 18:42

forsyth2 force-pushed the issue-48-v2-1-data branch from 8f2143b to 22fa84a Compare November 18, 2024 20:44

Updates on Perlmutter

d672c3c

forsyth2 force-pushed the issue-48-v2-1-data branch from 22fa84a to d672c3c Compare November 18, 2024 20:47

forsyth2 commented Nov 18, 2024

View reviewed changes

forsyth2 marked this pull request as ready for review November 18, 2024 21:05

chengzhuzhang reviewed Nov 18, 2024

View reviewed changes

chengzhuzhang requested changes Nov 18, 2024

View reviewed changes

Address comments

63226f1

chengzhuzhang approved these changes Nov 19, 2024

View reviewed changes

forsyth2 merged commit 40f78a2 into main Nov 19, 2024
1 check passed

forsyth2 deleted the issue-48-v2-1-data branch November 19, 2024 17:08

chengzhuzhang reviewed Nov 19, 2024

View reviewed changes

forsyth2 mentioned this pull request Nov 19, 2024

Fix CMIP6 line in v2.1 data #53

Merged

forsyth2 mentioned this pull request Apr 16, 2025

Add v2.1 BGC data #56

Merged


		The E3SMv2.1 simulation data is available on ESGF and NERSC HPSS.

		The preferred retrieval method is ESGF. Native output is available at `ESGF <https://esgf-node.cels.anl.gov/search/e3sm/?model_version=2_1>`_, and a subset of the data is reformatted to conform to CMIP conventions and submmited to the CMIP6 ESGF archive (ESGF links are provided in the table below for published data).


		v2_1.LR simulations data has been archived on NERSC HPSS under: ::

		/home/projects/e3sm/www/WaterCycle/E3SMv2_1

		/home/projects/e3sm/www/WaterCycle/E3SMv2_1


		Original run scripts can be found `here <https://github.com/E3SM-Project/e3sm_data_docs/tree/main/run_scripts/v2.1/original>`_.

		@@ -0,0 +1,10 @@
		model_version, group, resolution, category, simulation_name, machine, checksum, experiment, ensemble_num, cmip_only, node,


		or ::

		zstash extract --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/<HPSS path below>

Add v2.1 data #50

Add v2.1 data #50

Uh oh!

Conversation

forsyth2 commented Nov 13, 2024

Uh oh!

forsyth2 commented Nov 13, 2024

Uh oh!

chengzhuzhang commented Nov 14, 2024

Uh oh!

forsyth2 commented Nov 14, 2024

Uh oh!

chengzhuzhang commented Nov 15, 2024

Uh oh!

forsyth2 commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chengzhuzhang commented Nov 15, 2024

Uh oh!

forsyth2 commented Nov 15, 2024

Uh oh!

forsyth2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chengzhuzhang Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chengzhuzhang Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chengzhuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

forsyth2 commented Nov 19, 2024

Uh oh!

chengzhuzhang commented Nov 19, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

forsyth2 commented Nov 15, 2024 •

edited

Loading

chengzhuzhang Nov 18, 2024 •

edited

Loading

chengzhuzhang Nov 18, 2024 •

edited

Loading